Pothole and Wet Surface Detection Using Pretrained Models and ML Techniques

Abstract

Roads contribute significantly to the economy and serve as a transportation platform. Road potholes are a key source of worry in transportation infrastructure. The purpose of this research is to develop an Artificial Intelligence (AI) model for identifying potholes on asphalt pavement surfaces. Image processing techniques from pretrained models such as efficientnet, resnet50, mobilenet and ML models such as random forest, decision tree, SVC, SVM. Several studies have advocated employing computer vision techniques, including as image processing and object identification algorithms, to automate pothole detection. It is important to digitize the pothole identification process with acceptable accuracy and speed, as well as to deploy the procedure conveniently and affordably. Initially, a smartphone placed on the automobile windscreen captures many photographs of potholes. Later, by downloading pothole photographs from the internet, we expanded the amount and variety of our collection (2400 images with over 900 potholes). Second, to locate potholes in road photos, several object detection methods are used. To compare pothole detection performance, real-time Deep Learning algorithms in various setups are employed. Similarly Wet pavement decreases surface friction dramatically, increasing the likelihood of an accident. As a result, timely understanding of road surface condition is essential for safe driving. This research proposes a unique machine learning model pipeline for detecting pavement moisture based on live photos of highway scenes acquired via accessible to the public traffic cameras. We refined existing state-of-the-art feature extraction baseline models to capture background instance targets, such as pavement, sky, and vegetation, which are frequent in highway scenes, to simplify the learning job.

Introduction

I. INTRODUCTION

Roads serve as the foundation for people mobility and connecting diverse locations. The size of roadways varies according to  Its usefulness. Highways, for example, are big enough to accommodate several lanes built for heavy traffic. Nonetheless, roads within towns are built to be narrower and have one or two tracks. Roads are essential in people's everyday lives, thus they must be maintained on a regular schedule to maintain them operational and secure. Because of the large number of roads in a particular nation, it is impossible to maintain a constant evaluation of roads; hence, pothole creation cannot be predicted. The most common source of road issues is pavement deterioration. Road surface condition awareness is crucial for highway safety. Weather patterns, which have grown turbulent as a result of climate change, have a substantial influence on road surface quality. Wet pavement conditions diminish surface friction and raise accident risk.

Timely understanding of road surface conditions supports safer driving behaviours in both planning and navigation duties as well as real-time vehicle activities, including such braking system preemptive adjustments to minimise stopping distance. Preparatory changes to anti-lock braking systems (ABS) under poor weather conditions, for example, have been demonstrated to dramatically minimise stopping lengths, emphasising the necessity of evaluating existing road constraints in a reasonable timeframe.
II. PROBLEM STATEMENT

Mobility technology will continue to be vital and needs to be enhanced throughout time. In this day and age, there are several road expansions to offset the huge increases of automobiles. The increased number of automobiles generated issues including damaged roads and a lack of road maintenance.

The failure to fix damaged roads, particularly potholes, makes it more risky for cyclists to drive safely. This issue is becoming increasingly alarming as the number of accidents and deaths continues to rise. The pothole detecting sensor may be used on the automobile system to prevent accidents. The construction of a pothole detecting sensor in this study is based on the proximity sensor system, which employs a camera and a digital image method.

III.  Objectives

1. To detect refined pothole level statistics from image sifts.
2. To fill the gap of imbalance of constructor-pothole ratio.
3. The pothole endangers the safety of drivers. As a result, the construction of a real-time pothole detecting system for communicating pothole information may increase driving safety.
4. To improvise the performance and accuracy through large-scale datasets.
5. To offer straightforward output that is clear and easy to understand for everyone.

IV. LITERATURE SURVEY

Pothole And Wet Surface  Detection Using Pretrained Models And Ml Techniques plays a significant role in human beings daily lives. Keeping an appropriate braking distance, which is unpredictable and impacted by speed, weight, and road friction factor, is a critical component of driver safety. Anti-lock braking systems (ABS) in automobiles are tuned to function best on dry asphalt and are retrospectively modified to different surface conditions. In bad road surface conditions, this posterior adaptivity causes longer braking distances.

Transfer learning can save time by picking a pretrained model and fine-tuning it on the dataset. The object recognition API tensor flow model is employed in this case to detect potholes that may be used to recognise things in images. To identify potholes, advanced CNN models such as F-RCNN, Inception v2, and Inception v1 are utilised. After recognizing the pothole using the Android app, the motorist will be notified if there is a pothole on the road. R-CNN-based models have the disadvantage of taking longer to forecast. Additionally, models trained on roads in other nations for detecting potholes perform badly on Indian roads owing to substantially different damage situations.

Redmon  created the expression "YOLO" (you only look once). The input picture is divided into SxS grid cells, with each cell responsible for identifying an item and estimating its coordinates for the bounding box. Each item bounding box displays the X and Y dimensions, height (h), width (w), and confidence score, as well as the class title. The confidence score is the proportion of the actual labelled item bounding box that matches the bounding boxes box and indicates the correctness of the bounding box prediction. Some techniques need numerous scans of an input picture to identify, categorise, and locate many items in a single step. This technique, known as YOLOv1, was a watershed moment in real-time object identification.

Ping Pong A deep learning-based pothole detection approach is used to locate potholes on roadways by mounting a camera on the dashboard of a car and connecting it to the internet. The dataset from the classification process is utilised to train and assess CNN models such as Faster R-CNN, YOLOV3, SSD, and HOG with SVM. Data that is currently available will be converted into labelled image files for training datasets that will be used as input by the models. To produce satisfactory results, the hyper parameters for the model variables are modified to compute the size of a pothole. When compared to other models, the Yolov3 model performed well.

Using current infrastructure, successful road surface characteristics might be conveyed to vehicle aided driving systems. The Georgia Department of Transportation (GDOT), for example, now possesses around 2200 traffic monitoring cameras in the metropolitan Atlanta region that can be used for such purposes. To detect the presence of rain, Bossu et al. (2011) used image analysis on shuttered television (CCTV) video. Extensive picture data may be mined to serve as a stand-in for field sensors including such rain gauges. CCTV cameras are now commonly accessible in urban and suburban regions all over the world. Using deep learning in conjunction with the number of current cameras enables for extensive data to be easily gathered and used for a variety of applications.

Several models for segmenting road sceneries have been built using convolutional neural networks, although they have mostly focused on driver-view scenes at the ground level (Deng et al., 2017, Huo et al., 2019, Lyu et al., 2019). The goal of our research is to use advanced deep learning division architectures to comprehend road scenes taken by present Surveillance cameras.

V.    PROPOSED SOLUTION

The models depicted in Fig 2 use picture data as input parameters. Images are translated into a readable format and scaled to fit within the input model in the data preparation model. The dataset is separated into two parts: training data and testing data. Developing advanced CNN models with several layers and fine-tuning the parameters.

The models incorporated in the pretrained models were mobilenet, Inception, rest net 50, and efficientnet, and the accuracy of the above stated models were as follows: mobile net in epoch 1 accuracy was 82.13%, epoch 2 accuracy was 84%, and loss in epoch 1 was 46%, and loss in epoch 2 was 36%. Inceptionnet's accuracy in epoch 1 was 78.13% and 79.54% in epoch 2, with a loss in epoch 1 of 52.26% and a loss in epoch 2 of 47.34%.

Resnet50 accuracy in epoch 1 was 78.58% and 81.58% in epoch 2, with a loss in epoch 1 of 49.43% and a loss in epoch 2 of 43.67%. Efficientnet's accuracy in epoch 1 was 81.72%, whereas it was 83.4% in epoch 2.whereas the loss in the first was 44.65% and the loss in the second was 40.72%

Evaluation measures such as accuracy, precision, and recall are computed. Lastly, will save all in one file as h5, and then create the gui application and choose the photographs and model for predicting the output based on the selected model. Deep learning pretrained modules are used in the implementation.

The models depicted in Fig 2 use picture data as input parameters. Images are translated into a readable format and scaled to fit within the input model in the data preparation model. The dataset is separated into two parts: training data and testing data. Developing advanced CNN models with several layers and fine-tuning the parameters. Evaluation measures such as accuracy, precision, and recall are computed. Lastly, will save all in one file as h5, and then create the gui application and choose the photographs and model for predicting the output based on the selected model. Deep learning pretrained modules are used in the implementation.

A. Data Collection

The collection was compiled using high-resolution photos of motorway roads and muddy roads found on the internet. Pictures are acquired for one dataset from online sources, muddy roads dataset, and another dataset from Kaggle dataset. Plain and Pothole are the two labels assigned to images. The total collection comprises around 1000 photos

???????B. Data Preparation

The picture data used as input will be a large dataset with many photos. Resize the photos to 256*256 to convert them to a common format. The label encoder is used to convert all pictures. Defining simple and pothole trained data as model input.

???????C. Data Splitting

Data is classified into two types: training and testing. We must proportionally split the full data collection. For highway road datasets, 80 percent would most likely be utilised for training, while 20 percent are going to be employed for testing. For muddy road datasets, 60 percent will be used for training and 40 percent for testing.

The training dataset is a subset of the broader dataset from which the model is trained. A testing dataset is a subset of the training dataset that is used to put learnt models to the test.

???????D. Advantages of Proposed Solution

1. Deep learning models are used to forecast the accuracy of the characteristics being analysed .mobilenet, inception, resnet50, efficientnet are the models. First, we chose each model one by one and trained it with training data from that model. The trained model is stored as h5 after successful training.
2. The prediction is then performed with each model, and its accuracy is assessed using the test dataset.

VI. WORKING ENVIRONMENT

A. Hardware Requirements

1. PC, Mac or laptop with x86-64 (64-bit) compatible processors.
2. Operating System:  Linux, Windows, macOS
3.  RAM: 16 GB or greater recommended
4.  Storage: 500 GB SSD or greater recommended CPU:  2.1 GHz or faster Architecture: 32-bit or 64-bit

???????B. Software Requirements

1. IDE: (Any one of these)
2. VScode
3. Google Collab
4. Jupyter Notebook
5. Language: Python
6. Concept/ Technology: Deep Learning, CNN, ML,
7. Dataset (Trained & Tested): FER-2013
8. Libraries: NumPy, pandas, seaborn, matplotlib, cv2, os, TensorFlow, karas, Sklenarn-metrics, Conv2D, Dense, Batch Normalization, Activation, Dropout, MaxPooling2D, Flatten.

VII. METHODOLOGY

A. System Architecture

The ResNet-50 model is divided into five stages, each of which includes a convolution and Identity block. Every convolution block has three convolution layers, and every identity block has three convolution layers. There are about 23 million trainable parameters in the ResNet-50. To fine-tune the model, we utilised Dropout, three dense layers with Relu, and one thick layer with Softmax.

Inceptionresnet v2 mode use factorization to restrict channel size, hence reducing overfitting. The number of boundaries was also reduced, as an increase in the number of residual squares between beginning modules results in a large number of beginning modules. To fine-tune the model, we utilised Dropout, two dense layers with Relu, and one thick layer with Softmax.

Mobilenet is indeed an image recognition model that uses the input picture to generate the return frame and object class. This Single Shot Detector (SSD) object detection model makes use of Mobilenet as a backbone to deliver quick object identification that is optimised for mobile devices.

EfficientDet is an object detection model that employs many optimization and backbone changes, including the usage of a BiFPN and a compound scaling strategy that equally scales the resolution, depth, and breadth for all backbones, feature networks, and box/class prediction networks at the same time.

???????B. Classification and Detection

We completed all the preprocessing parts and the features are now more than ready to go to the next phase which is classification algorithms. We have applied four different algorithms of machine learning and all of them did a very good job at classifying and detecting the potholes. We are going to elaborate each of them down below.

1. Decision Tree: Decision Trees are types of directed machine learning in which information is separated by a predefined boundary. Two items, in particular decision nodes and leaves, can help to clarify the tree. This is one of the most popular machine learning algorithms. Despite the fact that this method provided the least accuracy. That didn't seem to operate properly with our system.  It provided us an 53% test set accuracy and a 57.72% validation accuracy.
2. Random Forest: This method is made up of numerous decision trees that have been bootstrapped together. It is a well-known algorithm. It gave us a real positive prediction accuracy of 100%, which is an incredible result, and a true negative prediction accuracy of 85%, which is a somewhat less anticipated outcome. This algorithm detects potholes with a test set accuracy of 59.87% and validation accuracy of 62.96%. 
3. SVC: Support Vector Classifier (SVC) is a supervised machine learning technique that is commonly used for classification problems. SVC works by mapping data points to a high-dimensional space and then determining the best hyperplane for categorising the data. Sklearn SVC is an SVC implementation supplied by the well-known machine learning toolkit Scikit-learn. It provided us with 55% accuracy for test sets
4. SVM: Support vector machine, is a common supervised machine learning technique. It is quite good at categorising and detecting. Our features have a prediction accuracy of 99% for genuine positives and 94% for true negatives, which is superior than any prior ones. It provided us with 55.46% accuracy for test sets and 55.45% accuracy for validation. Which is likewise not much better than the prior ones. 

IX. CHALLENGES

1. Priorities include object categorization and localisation. The first major challenge of object detection is its additional goal: we want to identify picture objects as well as establish their locations, which is known as the object localization problem. To solve this problem, researchers frequently employ a multi-task loss function that penalises both misclassifications and localization mistakes. It is difficult to decrease the amount of false positives in audio and image frames.
2. They must also be extremely quick at prediction time in order to fulfil the real-time needs of video processing. Numerous important improvements over the years have improved the performance of these algorithms, increasing test time from 0.02 frames per second (fps) for R-CNN to 155 fps for Fast YOLO.

X.  FUTURE SCOPE

1. Future work of this paper is to incorporate on even more data should be trained and tested in exchange to get the high accuracy with less error and boost the rate of prediction . For real-time detected objects, complexity and accuracy are all time considerations.
2.  To improve precision, a high-efficiency CPU might be employed. CNN architectures, such as the latest forms of origin (Xception and Mobile Net), are used to improve performance and precision.
3. In the future, instead of taking images, video can be captured. This provides a good picture of the pothole and allows for a more in-depth analysis of its severity.
4. Future work may also include the location of various anomalies such as Expansion joints, Manhole and Pipeline gaps, and so on. In the future, experts will be able to investigate strategies for determining the significance of an event at any vehicle operating speed.

IX. ACKNOWLEDGMENT

Without our supervisor, Dr. Kailas Patil, who has provided me with tremendous assistance, neither this work nor the study it is based upon would have been feasible. The kindness and knowledge of everyone have helped this study in countless ways and prevented me from making many mistakes; those that unavoidably remain are only our fault.

Conclusion

Finally, this work aimed to detect highway and muddy roads that are plain or have potholes using pictures and was analysed using a range of deep learning algorithms. Keras pothole picture has been learned and evaluated using tensor flow. The model was trained using around 3000 photos obtained from online sources, dataset Another dataset is drawn from the Kaggle(highway roads)dataset. The trained models are stored as h5.py for testing on the web application. The execution time varies depending on the complexity of the picture. The developed programme produces accurate and promising results on photos, both basic and complicated. This innovation has the potential to benefit self-driving applications and the automation sector. This study may be expanded to detect various pavement distresses, road depressions, categorise roads based on quality, and estimate pothole depth. Accuracy restrictions can be addressed in the future by further modifying and expanding the real-time deployment.

References

[1] Surekha Arjapure, D. R. Kalbande“ Road Pothole Detection using Deep Learning Classifiers” International Journal of Recent Technology and Engineering (IJRTE) ISSN: 2277-3878, Volume-8 Issue-6, March 2020 [2] Abhishek Kumar ,Chakrapani “A Modern Pothole Detection technique using Deep Learning” 2nd International Conference on Data, Engineering and Applications (IDEA), 19 August 2020. [3] Kanushka Gajjar, Theo van Niekerk, Thomas Wilm and Paolo Mercorelli, ”Vision-Based Deep Learning Algorithm for Detecting Potholes” preprints 10.20944/preprints202102.0569.v1, 25 February 2021. [4] Vijayalakshmi B, Kiran P, Kishore Jadav B, Madhusudhan G R, Manoj K S, ” Detection of Potholes using Machine Learning and Image Processing”, IJERTCONV8IS15039, 202 [5] K. Chang, J. Chang, and J. Liu, “Detection of pavementdistresses using 3d laser scanning technology,” in Proceedingsof the Computing in Civil Engineering (2005), pp. 1–11,Cancun, Mexico, May 2005. [6] Z. Hou, K. C. Wang, and W. Gong, “Experimentation of 3dpavement imaging through stereovision,” in Proceedings of theInternational Conference on Transportation Engineering 2007,pp. 376–381, Chengdu, China, July 2007. [7] P. Harikrishnan and V. P. Gopi, “Vehicle vibration signalprocessing for road surface monitoring,” IEEE Sensors Jour-nal, vol. 17, no. 16, pp. 5192–5197, 2017. [8] https://www.researchgate.net/publication/358977900_Pothole_Detection_Using_Deep_Learning_A_Real-Time_and_AI-on-the-Edge_Perspective [9] https://arxiv.org/ftp/arxiv/papers/2107/2107.06356.pdf [10] B. X. Yu and X. Yu, “Vibration-based system for pavement condition evaluation,” in Proceedings of the Applications of Advanced Technology in Transportation, pp. 183–189, Chicago, IN, USA, August 2006. [11] K. De Zoysa, C. Keppitiyagama, G. P. Seneviratne, and W. Shihan, “A public transport system based sensor network for road surface condition monitoring,” in Proceedings of the 2007 Workshop on Networked Systems for Developing Regions, pp. 1–6, Colombo, Srilanka, August 2007. [12] J. Eriksson, L. Girod, B. Hull, R. Newton, S. Madden, and H. Balakrishnan, “The pothole patrol: using a mobile sensor network for road surface monitoring,” in Proceedings of the 6th International Conference on Mobile Systems, Applications, and Services, pp. 29–39, Breckenridge, CO, USA, June 2008. [13] A. Glowacz, “Ventilation diagnosis of angle grinder using thermal imaging,” Sensors, vol. 21, no. 8, p. 2853, 2021.

Copyright

Copyright © 2023 Pratham Vernekar, Aniruddha Singh, Dr. Kailash Patil. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.